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Abstract 

In dynamical models of cortical networks, the recurrent con- 
nectivity can amplify the input given to the network in two 
distinct ways. One is induced by the presence of near-critical 
eigenvalues in the connectivity matrix W, producing large 
but slow activity fluctuations along the corresponding eigen- 
vectors (dynamical slowing). The other relies on W being 
nonnormal, which allows the network activity to make large 
but fast excursions along specific directions. Here we inves- 
tigate the tradeoff between nonnormal amplification and dy- 
namical slowing in the spontaneous activity of large random 
neuronal networks composed of excitatory and inhibitory 
neurons. We use a Schur decomposition of W to separate 
the two amplification mechanisms. Assuming linear stochas- 
tic dynamics, we derive an exact expression for the expected 
amount of purely nonnormal amplification. We find that 
amplification is very limited if dynamical slowing must be 
kept weak. We conclude that, to achieve strong transient 
amplification with little slowing, the connectivity must be 
structured. We show that unidirectional connections be- 
tween neurons of the same type together with reciprocal 
connections between neurons of different types, allow for am- 
plification already in the fast dynamical regime. Finally, our 
results also shed light on the differences between balanced 
networks in which inhibition exactly cancels excitation, and 
those where inhibition dominates. 



1 Introduction 

A puzzling feature of cortical dynamics is the presence of 
structure in spontaneously generated activity states. For ex- 
ample, activity in cat primary visual cortex fiuctuates along 
some non-random spatial patterns even when recordings are 
performed in complete darkness [T] Similarly, sponta- 
neously generated patterns of firing rates in rat sensory cor- 
tices occupy only part of the total space of theoretically 
possible patterns ^3 . As the constraints that govern these 
dynamics cannot be attributed to external stimuli, they are 
thought to originate from the patterns of synaptic connec- 
tivity within the network |3] [5] . This phenomenon is called 
patterned amplification. 



Patterned amplification can also be observed in simulated 
neuronal networks, in which spontaneous activity can be 
modelled as the response to unspecific, noisy inputs deliv- 
ered to each neuron individually. Propagated through recur- 
rent connections, these noisy inputs may cause the activity 
of some neurons to transiently deviate from their average 
more strongly than could be expected from the variability 
of the external inputs. We thus define amplification here as 
the strength of these additional, connectivity-induced fluc- 
tuations. 

Let us consider the following simple linear model for 
stochastic network dynamics: 

dx^ —{VJ -l)x + a^d$, (1) 



where r is the neuronal time constant, x e 



is the de- 



viation of momentary network activity with respect to a 
constant mean firing rate, W is an A/ x A/ synaptic con- 
nectivity matrix, 1 is the identity matrix, and is a noise 
term modelled as a Wiener process. The fluctuations of X;(t) 
around zero (i.e. around the mean firing rate of neuron /) is 
caused by the noisy input and the recurrent drive. Starting 
from arbitrary initial conditions, the network activity x con- 
verges to a stationary Gaussian process with covariance ma- 
trix Z = {cy} (at zero time lag), provided no eigenvalue of 
W has a real part greater than unity. This covariance matrix 
has a baseline component Hunc. — (J^t1/2 that corresponds 
to the covariance matrix in the absence of network connec- 
tions ("unconnected"). Wiring up the network yields addi- 
tional correlations and gives potentially rise to larger fluc- 
tuations of the activity of individual units. We define this 
amplification A as the ratio [Tr(Z) — Tr(Zunc.)] /Tr(5Iunc.)- 
In other words, A measures the relative gain in mean vari- 
ance that can be attributed to the recurrent connections. 
That is, 

A/ 



A{\N) 



dcf 



TCr|A/ 



1 



(2) 



Under linear dynamics like that of Eq. [T] amplification 
can originate from two separate mechanisms. A first, "nor- 
mal" type of amplification can arise from eigenvalues of W 
with real parts close to (but smaller than) 1. The noise ac- 
cumulates along the associated eigenvectors more than in 
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other directions, giving rise to larger activity fluctuations 
and substantial dynamical slowing along those axes. If the 
synaptic connectivity is normal in the mathematical sense 
(WW^ = W^W), it is the only mechanism through which 
the network can amplify its input [S]. Indeed, if W is nor- 
mal, its eigenvectors form an orthonormal basis. The sum 
of variances in this eigenbasis is therefore equal to the sum 
of variances of the neuronal activities in the original equa- 
tions. Since linear stability imposes that every eigenvalue of 
W has a real part less than one, the activity along the eigen- 
vectors can only decay following some initial perturbation. 
In other words, a stable normal linear system is contractive: 
no initial condition can transiently be amplified. If the ma- 
trix W is not normal (WW^ ^ W^W), another, nonnormal 
type of amplification can also contribute |5l |6l [ll [8] . The 
eigenvectors are no longer orthogonal to each other, and the 
apparent decay of the activity in the eigenbasis can hide a 
transient growth of activity in the neurons themselves. Such 
growth can only be transient, for stability requirements still 
demand that the activity decay asymptotically in time. 

Purely nonnormal amplification that does not rely on dy- 
namical slowing may be ideally suited for sensory cortices 
that need to track inputs varying on fast timescales (possibly 
of order r). It has also been identified as a key mechanism 
for short-term memory of past inputs, for in certain circum- 
stances, hidden feedforward dynamics enables the network 
to retain information about a transient stimulus for a dura- 
tion of order A/r |7]. The presence of noise as in Eq.[l] could 
limit this memory duration to -s/A/t [SI [9], but this is still 
much longer than the time r in which individual neurons 
forget their inputs. 

The above considerations apply to purposely structured 
networks [UlTll^. It is not clear, however, how much of this 
beneficial kind of amplification can be expected to arise in 
randomly connected networks of excitatory and inhibitory 
neurons, a ubiquitous model of cortical networks. Murphy 
and Miller [5 convincingly argued that nonnormal ampli- 
fication should generally be a key player in the dynamics 
of balanced networks, i.e. when strong excitation interacts 
with equally strong inhibition and when neurons can be only 
excitatory or inhibitory but not of a mixed type. When the 
connectivity is dense, or at least locally dense, weak patterns 
of imbalance between excitation and inhibition can indeed 
be quickly converted into patterns in which neurons of both 
types strongly deviate from their mean firing rates. Here, 
we revisit nonnormal amplification in the context of ran- 
dom balanced networks. We derive an analytical expression 
for the purely nonnormal contribution to amplification in 
such networks. The analysis reveals a strong tradeoff be- 
tween amplification and dynamical slowing, suggesting that 
the connectivity must be appropriately shaped for a net- 
work to simultaneously exhibit fast dynamics and patterned 
amplification. 



2 Separating the effects of normal and 
nonnormal amplification 

In the introduction, we have distinguished normal from non- 
normal amplification. The Schur decomposition (Fig. [T]) - 
a tool from linear algebra - offers a direct way to assess 
the contributions of both mechanisms to the overall amount 
of amplification A{\N). Any matrix W can be written as 
(A + T) U where U = {t///} is unitary, A is a diagonal 
matrix that contains the eigenvalues of W, and T — {ty} 
is strictly lower-triangulai[^ (Fig. [l^^c) . The lines of U are 
called the Schur vectors (or Schur modes) and are all orthog- 
onal to each other. If this decomposition is to avoid complex 
numbers, A is only block-diagonal, with 2x2 blocks contain- 
ing the real and imaginary parts of complex conjugate pairs 
of eigenvalues, and 1x1 blocks containing the real eigenval- 
ues. Importantly, because the Schur basis U is orthonormal, 
the sum of variances in the basis of the Schur vectors is equal 
to the sum of the single neuron activity variances. Thus, in 
order to compute >4(W), one can instead focus on the ac- 
tivity fiuctuations in an abstract network whose units cor- 
respond to spatial patterns of neuronal activity (the Schur 
vectors) and interact with a connectivity matrix A + T (Fig. 
[l^, right). This matrix is lower-triangular, so the abstract 
network is effectively feedforward. In the Schur network, 
unit / receives its input from all previous units j < i accord- 
ing to the row of T. Without input, the activity of unit 
/ decays at a speed governed by eigenvalue A/. 

A network with a normal connectivity matrix would have 
only self-feedbacks (T — 0), thus being equivalent to a set 
of disconnected units with a variety of individual effective 
time constants, reflecting dynamical slowing or acceleration. 
Amplification-by-slowing therefore arises from A (Fig. [T|3), 
which summarises all the "loopiness" found in the original 
connectivity. Conversely, when A = 0, all units share a com- 
mon time constant r (which is also the time constant of the 
actual neurons) and interact in a purely feedforward man- 
ner via matrix T (Fig. [l|:) . We refer to this case as "purely 
nonnormal" , because the network is then free of the unique 
dynamical consequence of normality, namely a modification 
of the speed of the dynamics "Purely nonnormal" am- 
plification therefore arises from matrix T that reveals the 
functional feedforward connectivity hidden in W. 

The latter situation (A = 0) is the focus of this paper. By 
substituting W with T in Eq. [l] and subsequently calculat- 
ing A{T) as defined in Eq. [5] we intend to reveal the frac- 
tion of the total amplification /4(W) in the neuronal network 
that cannot be attributed to dynamical slowing, but only 
to transient growth. This constitutes a functional measure 



^ Upper, not lower, -triangular T is more common in the literature, 
but we prefer to keep the flow of information forward (from the I''' to 
the /V"' Schur mode) for notational convenience in our calculations. 

■^Note that quantifying nonnormality can be done in a variety of 
ways, e.g. through several measures of "departure from normality" [H]. 
Our concept of "pure nonnormality" is therefore more specific to our 
particular purpose, in that it expresses the absence of normal effects 
on the dynamics of the neurons. 
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of nonnormality. We carry out this analysis in a statistical 
sense, by calculating the expected amount of purely nonnor- 
mal amplification {A(T)) where the average (•) is over the 
random matrix W. In section |3j the ensemble statistics of 
W are defined, and related to the statistics of the non-zero 
entries of T. In sections |4] and [Sj we perform the calculation 
of {A{T)). 

3 Schur representation of neural connectivity 

matrices 

Prior to calculating the nonnormal contribution to amplifi- 
cation in realistic neural connectivity matrices, we first anal- 
yse the statistical properties of the Schur triangle T derived 
from a neuronal network where every pair of neurons has 
a certain probability of being connected in either direction. 
Specifically, we consider networks of N /2 excitatory and N /2 
inhibitory neurons, with connectivity matrices W drawn as 
follows (Fig. [1)1): 



(a) original W 



1 



Wii 



N 



+ W0 if j < N/2 
-Wo if j > N/2 
with proba. (1 — p) 



with proba. p 



(3) 



Excitation and inhibition are thus globally balanced. The 
1/V~N scaling ensures that in the limit of large A/, the eigen- 
values {Xk} of W become uniformly distributed inside the 
disk of radius 

R = 



Wq 



Vp(i - P) 



(4) 



and centered around zero in the complex plane (Fig. [T|d), 
with the exception of a few outliers [1^ . To push the outliers 
inside the disk, we enforce that excitatory and inhibitory 
synapses cancel each other precisely for each receiving neu- 
ron, i.e. Wv = with V = (1, 1, • • • , 1)/VA/ tH HI]- This 
constraint is also essential to the identification of the ensem- 
ble statistics of T as detailed below. Such a "global balance" 
can be achieved by a Hebbian form of synaptic plasticity at 
inhibitory synapses in random spiking networks |12j . Here 
we enforce it by subtracting the row average (a small num- 
ber) from every row (which accounts for the barely visible 
horizontal stripes in W of Fig. [l^). 

The main point in relating the statistics of T to that of 
W is to note that the Schur basis is unitary, so that the sum 
of squares in W is also equal to the sum of squares in A + T. 
Thus 



l<iJ<N 



E 

l<k<N 



(5) 



From our choice of the weights wij (Eq. w| and assuming 
that N is large enough, we can derive — NpwQ. Fur- 

thermore, knowing that the eigenvalues lie uniformly inside 
the disk of radius /?, we can write ^ \Xk\^ ~ NR^/2 which is 



Schur basis U 




eigenvalue 
spectrum A 



Schur T 

(strictly lower 
triangular) 
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Figure 1: Teasing apart normal and nonnormal amplification in ran- 
dom networks of excitatory and inhibitory neurons, (a) Example 
sparse neural connectivity matrix W (left, 50 exc. columns and 
50 inh. columns, thinned out to 30 x 30 for better visibility), a 
schematics of an associated Schur basis U (center), and the corre- 
sponding abstract network of Schur modes, in which the interactions 
are feedforward from top to bottom (right). The Schur vectors in U 
(center), orthogonal to one another, represent patterns of neuronal 
activity in the original network. The last Schur vector is explic- 
itly chosen to be the uniform "DC" mode v = (1, 1, ■ ■ ■ , 1)/-\/A/ 
and is represented here in green, (b) Amplification via dynamical 
slowing ("normal" amplification) is described by the set of eigenval- 
ues A = (Ai, ... , A/v) of W, which for a random network lies inside 
a disk centered around zero in the complex plane. These eigenval- 
ues determine the decay rates of the Schur patterns, (c) Nonnormal 
amplification arises from the strictly lower-triangular matrix T which 
describe the purely feedforward part of the interactions between the 
Schur patterns. The first non-zero entry in the upper left corner of 
T is t2i and represents the "forward" coupling from the first Schur 
mode onto the second. The last row (f/vi, t/v2, ■■. , («(«-!)). zoomed- 
in (green arrow), is the coupling from the first A/ — 1 Schur modes 
onto the last (uniform) Schur mode v. (d) For a fixed large matrix 
W, the non-zero entries t,j in matrix T are approximately normally 
distributed with zero mean and variance given by Eq. [9] (black his- 
togram, for j < i < N). The entries in the last row have larger 
variance given by Eq.[8](/ = N, green histogram), (e) Moreover, 
the variance (t,y) across many realisations of W is the same for all 
j < i < N (black histogram). Similarly, {tfij) is the same for all 
j < N (green histogram), (f) The correlations {tijtki) (for / 7^ /c or 
j 7^ I) are negligible, as seen from a comparison of their empirical 
distribution (black) with surrogate data from triangular matrices in 
which non-zero entries are all drawn i.i.d. (grey, barely visible un- 
der the black curve). The data for panels (d-f) was acquired by 
Schur-transforming 5,000 random weight matrices of size N = 100, 
drawn as described in section [3] with connection density p = 0.1 and 
spectral radius /? = 1. 



''it is straightforward to allow for any distribution of non-zero 
weights; as it turns out, this Dirac delta distribution achieves max- 
imum nonnormal amplification. 



also valid for large N. We replace these sums in Eq. [5j sim- 
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plify the result using Eq. [4] and obtain the overall empirical 
variance of the non-zero entries in T, to leading order in N: 



N{N-1) 



/?2 1 



N 1 



(6) 



Note that this empirical variance is not necessarily equal to 
the ensemble variance (t,^) — for fixed / and j. In fact, 
we have observed that if the non-unique Schur basis is chosen 
arbitrarily, ( t?j'j computed over many realisations of W is not 
uniform across rows, but rather tends to increase with row 
index /. This heterogeneity is difficult to characterise, and 
undermines the calculation of amplification developed in the 
next section. Fortunately, we can circumvent this problem 
by choosing the uniform eigenvector v of W as the last Schur 
vector: UNk = 1/ V~N for all /j^ Coefficient tjj then becomes 
distributed with the same zero mean and variance for all 
j < i < N, with the exception of the tA// coefficients which 
have higher variance (black and green lines in Fig. 
and[l^, empirical observation). Note also that the ensemble 
pairwise correlations between coupling strengths tij and t^i 
with / ^ j OT j ^ £ seem negligible (Fig. [l]f ) . 

We now proceed in two steps. First, we focus on the 
variance of the elements in the last row of the Schur matrix 
T, and then we turn to all the other non-zero components. 



To calculate variance Q 
and write for j < N 



^Nj ' definition of T 



N N 

1=1 k=l 

N / N 



(7) 



m^X^r)"" 

To leading order in N we can write w^i — ±pwQy/N 
where the ± sign depends on £ being smaller than N/2 (+, 
excitatory) or greater (— , inhibitory) - see Fig. [l^. For 
j < N, the J**" Schur vector Uj is orthogonal to the last 
Schur vector v cx (1, 1, ... , 1), so its components strictly sum 
to zero: J^e ^je — 0- Moreover, because of the normaliza- 
tion, J2e 'Jje — l- We can therefore approximate uji by a 
stochastic process with zero mean and variance 1/N. As- 
suming the Uji are uncorrelated, the variance of ti^j is thus 
simply WqP^ to leading order, which according to Eq. |4]is 
also 



Co 



(8) 



Notably, the variance Co in the last row of coupling matrix T 
is of order 1, and depends super-linearly on the connectivity 
density p (Fig. [2j green lines). 

We now turn to the other rows / < A/ of the Schur 
matrix T. Because all components tjj for j < i < N 



*This is always possible, since a Schur basis can be constructed 
through Gram-Schmidt orthonormalisation of the eigenbasis of W, so 
choosing v to enter the process first results in v being the last vector 
in a basis that makes W iotyer-triangular 




50 100 200 400 

N 

Figure 2: Linking the Schur triangle to the parameters of the neural 
connectivity matrix, (a) The variance of the entries in the strict 
lower triangle T scales linearly with the square of the spectral radius 
of the original weight matrix W. For the last row of T, the 
slope of depends on the connection probability p (top plot). For 
the rest of T, C,^ depends only on (bottom plot). Each point 
was obtained by empirically estimating (^^ and Co from 10 different 
Schur-transformed random neural weight matrices of size N = 400. 
Lines denote the analytical expressions in Eqs. [s] and [o] (b) Co in 
the last row of T scales super-linearly with the connection density 
p (top plot). In contrast, C^ does not depend on p (bottom plot), 
(c) In the last row of T, the variance is network size-independent 
(green line). In the rest of T, the variance is inversely proportional 
to N (black line, note the log-log scale). 



seem to come from the same distribution and look uncor- 
related (Fig. [TJl— f ), the empirical estimate of their variance 
2 J2j<i<N ^fj/i^^^)i^^^) coincides with the ensemble vari- 
ance C^ = ( / so long as N is large enough. Similarly, we 



can write J2j ^lij/i^ 1) = Co- Thus, the l.h.s. of Eq. [e] 
becomes C,^ + 2(q/N to leading order in N. Using Eqs.jojand 
|8]we conclude 



R2 



(9) 



Figure [2] shows that Eqs. [8] and [9] provide a good match to 
numerical results. 

At this point we can already draw a few conclusions. Sup- 
pose each unit in our Schur network receives external input 
of variance 1. First, since the uniform mode v receives net- 
work input from the remaining N — 1 Schur patterns with 
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coupling coefficients of order 1 (Eq. |8]) , we expect the global 
("DC") population activity x • v to fiuctuate macroscopi- 
cally, i.e. with a variance of order N. In contrast, the rest 
of the Schur modes should display fiuctuations of order 1. 
Second, we directly see that making the network denser (i.e. 
increasing p) can only result in larger DC fiuctuations, but 
no further amplification of the other (zero-mean) Schur pat- 
terns. This is because (q, but not C^, depends on p. Third, it 
is easy to see where these large DC fluctuations would orig- 
inate from. Imagine breaking the overall exc.-inh. balance 
in the network activity by a small amount, e.g. by initial- 
ising the network state x to d = (1 1, — 1 — l)/-\/A/, 

where we emphasize the scaling in l/^/~N. According to Eq. 
[l] the transient response to this perturbation is roughly Wd, 
which to leading order in N equals 



Wd 



pwo(l, 1, • • • ,1) 



(10) 



We note that the 1/vA/ scaling is gone. Thus, the network 
responds to a microscopic global balance disruption - a state 
in which the deviation between the excitatory and inhibitory 
population firing rates is of order I/VN - by an excursion of 
order 1 in the combined firing rate of both populations (see 
[5] for a more in-depth discussion of this effect). Finally, 
it is instructive to see what happens when the fun ctio nal 
feedforward link from d to 
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IS 



' A/ • V - expressed in Eq, 
removed from W. This can be achieved by transforming W 
into W' given by 



w = w 



PWQ 

A/ 



(I,-- - 



,1,-1, 



,-1) (11) 



It is easy to see that W'd = 0. In this case, calculations sim- 
ilar to Eqs.[5}j8]yield = ^ R^/N so that the DC fiuctu- 
ations are back to order 1: the amplification along the DC 
mode becomes comparable in magnitude to the amplifica- 
tion that occurs along any other Schur directions. Note that 
the operation in Eq.[Tl] effectively shifts the mean excitatory 
(resp. inhibitory) weight from pwq/^/N (resp. —pwo/\/l\i) 
to zero. We now substantiate these preliminary conclusions 
through a direct calculation of nonnormal amplification. 

4 Amplification in random strictly triangular 

networks 

We have seen in the preceding two sections that a randomly 
coupled network of excitatory and inhibitory neurons can 
be transformed via a unitary Schur basis into a different 
network where the couplings between units are given by a 
lower triangular matrix (Fig.jl^). Furthermore, the "purely 
nonnormal" part of the amplification of the external noisy 
input in the original network of neurons corresponds to the 
activity fiuctuations in the new feedforward network where 
all self-couplings are neglected (Fig. [T]:). Finally, we have 
also seen that it is possible to constrain the Schur basis 
such that the couplings between the first N — 1 units in the 
feedforward network are independently distributed with the 



same zero-mean and a variance given by the parameters of 
the original synaptic weights (Eq. |9]). In this section, we 
therefore study this "canonical" case, starting directly from 
a strictly lower-triangular matrix T and ignoring - for the 
moment - the transformation that gave rise to T. 

We want to solve for the expected variances of A/ ^ 1 
Ornstein-Uhlcnbeck processes (as in Eq. [T]) coupled by a 
strictly lower-triangular weight matrix T (therefore describ- 
ing a purely feedforward network, see inset in Fig. [3^) . We 
assume all non-zero coupling strengths to be sampled i.i.d. 
from some common distribution with zero mean and vari- 
ance a^/N. Due to the coupling matrix, the fluctuations 
that the external input causes in the first unit feed and aug- 
ment those it causes in unit 2. The third unit in turn fluc- 
tuates due to the external input and the activities of units 
1 and 2, and so on. We therefore expect the activity vari- 
ance an in unit / to increase with index /. In appendix [Xj 
we show that in the limit of large N and for some fixed 
< X < 1, the relative expected variance of the activity in 
unit / — xN is g{i/ N) = 2 {an) /Ta^ where the function g{x) 
is lower-bounded in closed form by 



1 

3 + \/3 

2H-73 
3-f 73 



exp 



exp 



1 - V3 
4 

l + \/3 



(12) 



a^x 



(Fig. [3] dashed blue curves) . We also derive the exact solu- 
tion as a power series 



r{x) = lim V/3/cX'' 



(13) 



k=0 

with the j3k coefficients defined recursively as 
^0 = 1 

„2^ " 



2k\ 



E 

i=0 



e\ (e + i)\ 



P. 



(14) 



The overall amplification y4o(a^) in the network is subse- 
quently obtained by integrating this variance profile g(x) 
from to 1, which corresponds to taking Eq.[2]to its A/ —> 00 
limit: 



k=0 



1 



- 1 



(15) 



Figure |3] shows that Eqs. 13 and [15] indeed converge to the 
empirical mean variance profile and mean amplification as 
the cut-off parameter K of the power series becomes large 
(red lines, K = 10). Figure [SjD furthermore shows how am- 
plification explodes with the variance cP-jN of the feedfor- 
ward couplings in the network. 

5 Amplification in random balanced networks 

Using the canonical result of the previous section that is 
restricted to homogeneous random lower-triangular matri- 
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(a) 




in W: 



0.25 0.5 0.75 
Schur mode x = i/N 




a 

Figure 3: Analytical result for a feedforward network of N Ornstein- 
Uhlenbeck processes coupled via a random strictly lower-triangular 
matrix (inset), (a) The expected activity variance (an) accumulates 
super-linearly from the first unit to the last down the feedforward 
chain. Dashed blue lines depict the closed-form lower-bound of Eq. 
|12| Red lines denote the exact solution given in Eq.jlS] truncated 
to K = 10. Open circles represent the numerical solution of Eq.[l]- 
or more exactly, the numerical solution of equation [20] given in the 
appendix - averaged over 20 randomly generated matrices of size 
N = 500. Each matrix T is characterised by the variance oP' / N of 
the coupling coefficients t,j with j < i. The strength of the external 
noise driving each unit independently is set to (7| = 2/r so that all 
activity variances in the network would be 1 should the couplings 
t,j be set to 0. (b) The total amplification (the area under the 
curves in (a), minus 1) explodes with increasing variance a^/A/ in 
the triangular connectivity matrix. Points and lines have the same 
meaning as in (a). 



ces, we now calculate A{R, p) = {A{T)} with T originating 
from the Schur decomposition of a neuronal connectivity 
matrix as in section |3j with connection density p and spec- 
tral radius R. Equation |13| c an directly be applied with 
a^/N = = R^/N (see Eq.|9| to describe the activity fluc- 
tuations of the first A/ — 1 Schur modes. The last Schur 
unit, however, receives feedforward input with couplings of 
variance ^ (Eq. [s]). Consequently, the expected vari- 
ance ((Ta/a/) of its temporal fluctuations has to be treated 
separately. In appendix IS] we show that 



lim 



N 



1 



(16) 



where g is given by Eqs. [T3|and[T4{ here with a = R. Gath- 
ering the contributions of all Schur modes, we obtain the 
expected overall amount of purely nonnormal amplification 



A{R,p) = Ao{R^ 



(17) 
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with Ao{R^) given by Eq. 

Figure shows that the nonnormal contribution to am- 
plification in the neuronal network explodes with the spec- 
tral radius R of the connectivity matrix W. This is because 
the amplification of the first A/ — 1 Schur units explodes 
with the variance of their feedforward interactions (Fig. 
3)3) and that is directly related to R (Eq. [9|. Note that 
for /? > 1 (to the right of the dashed vertical line in Fig. 
4^), the network of neurons is unstable. Although the con- 
cept of amplification in an unstable network is ill-defined, 
the "purely nonnormal" part of the total (infinite) ampli- 
fication remains bounded. Indeed, the purely feedforward 
network T derived from the Schur decomposition of W is 
itself always stable, since zero is the only eigenvalue of T. 
The instability in W arises from purely normal effects, when 
the real part of one eigenvalue of W exceeds unity so that 
dynamical slowing becomes infinite. 



Equation 16 confirms what we had previously discussed 
at the end of section [3} the last Schur unit has temporal 
fluctuations v • x(t) of variance 0{N). Those fluctuations 
thus make up for a finite fraction of the total nonnormal 
amplification (the last term in Eq. 17) as A/ — 00. Because 



the last Schur vector is the normalised uniform spatial pat- 
tern (1, ... , 1)/\/A/, the variance of the overall population 
activity /i(t) ee X^^KO/'^ = \/A/(x • v(t)) is of order 1. As 
we had foreseen in section [s] one can restore the 1/ N scal- 
ing of the these "DC" fluctuations (/i^(t)) by performing 
the operation of Eq. [TT] on the connectivity matrix W, i.e. 
subtracting a common constant from all excitatory weights 
(including zero weights) to make sure that they average to 
zero, and adding the same constant to all inhibitory weights 
with the same purpose. This situation is depicted by the 
grey curves in Fig. |4] Figure |4)3 shows that only these DC 
fluctuations depend on the connectivity density p. 

Overall, Fig. allows us to draw two important conclu- 
sions. On the one hand, if the level of dynamical slowing 
is to be kept low {R ^ I), only modest levels of amplifi- 
cation can be achieved (see the small amount of nonnor- 
mal amplification on the l.h.s. of the dashed vertical line). 
For example, if no mode is to decay with more than twice 
the single neuron time constant (Re(A) < 1/2), the average 
variance cannot exceed that of a disconnected network by 
more than 10%. On the other hand, the nonnormal con- 
tribution to amplification explodes with increasing R, i.e. 
with increasing synaptic strengths if the connection density 
is taken fixed. This suggests that strong transient amplifi- 
cation without dynamical slowing can only be achieved in 
structured, "less random" networks. The structure must al- 
low the synaptic couplings to assume larger values without 
causing the eigenvalue spectrum of W to reach instability. 
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(a) (p = o.i) 




0.5 1 1.5 

spectral radius R 




no 'DC 



0.1 0.2 0.3 0.4 
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Figure 4: Nonnormal amplification in random neuronal networks, 
(a) The mean amount of purely nonnormal amplification (/4(T)) = 
A{R, p) is reported as a function of the spectral radius R of W. 
Open circles denote the numerical solution of Eq. |20] averaged over 
20 randomly drawn connectivity matrices with connection density 
p = 0.1 and size N = 500. Errorbars denote the standard deviation 
over all trials. Red lines depict the exact solution in Eq.[l7] The grey 
line and grey circles indicate the mean removal of Eq. |ll| applied to 
W, which effectively removes the global macroscopic fluctuations of 
the entire population (labelled "no DC"). The dashed vertical line 
represents the limit of linear stability, beyond which the nonnormal 
part of amplification is still well-defined, (b) Same as in (a), now as 
a function of the connection density p for a fixed R — 1. In both (a) 
and (b), parameters p and R fully determined the value ±wo/V~N 
of the nonzero synaptic weights as wq = p(l — p) (cf. Eq.|4]|. 

6 DifFerent numbers of excitatory and 
inhibitory neurons 

We now consider the biologieally more plausible case of dif- 
ferent numbers of excitatory and inhibitory neurons. Typ- 
ical models of cortex assume fN excitatory neurons and 
(1 — f)N inhibitory neurons with f — 0.8 or similar. In 
this case, the eigenvalues A are no longer uniformly scat- 
tered inside the disk of radius R in the complex planc[^ but 
become more concentrated in the middle following a radially 
symmetric density /5(|A|) known analytically from TDJ (Fig. 

insets). As before, we consider the case where excitatory 
(resp. inhibitory) synaptic couplings are with probability 



^Rajan and Abbott showed that this happens when the variances 
of the excitatory and inhibitory weights differ (the variances comprise 
both the zero and non-zero synapses). Decreasing the number of in- 
hibitory neurons in a balanced network requires the strength of inhibi- 
tion to be increased. In sparse networks like ours, this automatically 
makes the overall variance of the inhibitory synapses larger than that 
of excitatory synapses, hence the observed effect on the eigenspectrum. 
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Figure 5: Networks with different numbers of excitatory and in- 
hibitory neurons, (a) Nonnormal amplification as a function of the 
spectral radius R of W, in sparse random balanced networks with 
fN excitatory and (1 — f)N inhibitory neurons, for f = 0.5 (black) 
and f = 0.8 (red). The connection density p was set to 0.1. The 
dashed vertical line represents the limit of linear stability, beyond 
which the nonnormal part of amplification is still well-defined. Solid 
circles were obtained by averaging the numerical solution of Eq. |20| 
for 20 random matrices of size N = 500. Errorbars denote standard 
deviation over all trials, (b) Filled circles show the scaled variance 
N (^tfj") / R^ of the non-zero Schur couplings in row / as a function of 
//A/ and for three different values of f. These variances were com- 
puted by Schur-transforming 100 matrices of size N = 200, with 
R = 1 and p = 0.1. Cyan lines denote the density p of eigenvalues 
A inside the unit disk [lOj, as a function of (1— |A//?|)^. Insets show 
the eigenvalue spectra of three example matrices of size N = 1000. 



(1 — p), and +we/\'N (resp. —wj/vN) otherwise. The 
global balance condition reads fwE — [1 — f)wi . To impose 
a given spectral radius R, we set w| — Wq{1 — f)/f and 
wf = w^f/{l - f) with = /?Vp(1 - P)- 

The results of section [3] regarding the variances in the 
Schur triangle have to be adjusted to accommodate these 
modifications. The derivation of Cq is left unchanged, so 
that the couplings t^j onto the uniform mode v still have 
the variance given by Eq. [Sj which notably does not depend 
on f . Using Eq. [s] we can then write down the empirical 
variance in the first N — 1 rows of T as 



N{N-1) 



E 

J<i<N 



N 



rp{r)dr 



(18) 
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Unfortunately, the ensemble variance (t?) for fixed /' and J is 
in general different from the average across matrix elements 
given by Eq. ( [T8| . Indeed, contrary to the case f = 0.5 
considered in section |3j the non-zero elements of T no longer 
have the same ensemble variance. Instead, (t?) grows with 
row index / < A/, and this profile interestingly matches the 
density of eigenvalues p |^ according to 



N 



R 1 




for j < i < N (19) 



This is depicted in Fig. [5)d. 

In a feedforward network like that of Schur units con- 
sidered here, a good strategy to generate greater amplifi- 
cation would be to give comparatively more power to the 
couplings onto earlier nodes. This is because amplification 
builds up super linearly along the feedforward chain (Fig.jsj), 
so that boosting early nodes exacerbates the avalanche ef- 
fect (see also j6]). Setting f to more than 0.5 does precisely 
the contrary: couplings onto early nodes become compara- 
tively smaller in magnitude, as shown by the filled circles in 
Therefore, simply replacing / N in Eq. 15 by the 



Fig.|5)[ _ 

empirical variance of Eq. [18] yields an over- estimation of the 
true amplification in the first N—1 Schur units (compare the 
red line with the red circles in Fig.lsk). We found it difficult 
to incorporate this variance profile {tjj') into the derivation of 
appendix]^ so we can only consider as accurate the results 
of numerical simulations. 

The conclusions reached at the end of section [5] do not 
change significantly under the more realistic assumption of 
f = 0.8. Although amplification almost doubles relative 
to f — 0.5, it remains very weak in the stable regime (to 
the left of the dashed vertical line in Fig. [5^), confirming 
that amplification can only come with substantial dynamical 
slowing when connections are drawn at random. 

7 Example of network structure for 
nonnormal amplification 

Here we show that random networks can be minimally struc- 
tured in such a way that strong nonnormal amplification oc- 
curs already in the fast dynamical regime. We exploit the 
fact that correlations in the connectivity matrix can mod- 
ify the shape of the eigenvalue spectrum. Symmetrising (or 
anti-symmetrising) W has been shown to generate elliptical 
(as opposed to circular) eigenspectra, in the case of "cen- 
tered" matrices where the distinction between excitatory 
and inhibitory neurons is not made |13| . Here we consider a 
modification of the sparse neural matrices studied in section 
[3] that achieves this slimming effect in the case of balanced 
networks (see the insets inFig.|6ji). AU non-zero entries as- 
sume a value ±wo/\/l\l, the sign depending on the excitatory 



^This happens provided the eigenvectors are sorted in decreasing or- 
der of their corresponding eigenvalue moduli, prior to going through the 
Gram-Schmidt orthonormalisation process. This results in a unique 
Schur basis. 
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Figure 6: Example of a network structure that favors nonnormal am- 
plification: unidirectional vs. bidirectional synaptic connections, (a) 
We varied the degree of anticorrelation between reciprocal weights 
in the connectivity matrix, as the fraction k of the maximum value 
it can assume, which is dictated by the connection sparsity (see 
text). This caused the eigenspectrum to stretch more and more 
along the vertical axis (compare the two insets), effectively decreas- 
ing the spectral abscissa R' (black filled circles). Empirical data was 
obtained from numerically computing the eigenvalues of 20 differ- 
ent matrices of size N = 500. Errorbars denote standard deviations 
over all trials. Gray lines are linear fits, (b) Nonnormal amplifica- 
tion as a function of the spectral abscissa R' . When all connections 
between an excitatory (E) and an inhibitory (I) cell are made recip- 
rocal, while all E — )■ E and I — > I connections are kept unidirectional 
(orange curve, corresponding to «; = 1 in (a)), stronger amplifica- 
tion is obtained in the fast dynamical regime (/?' <C 1). The black 
curve is here reproduced from Fig.[4](purely random case, k = 0) for 
comparison. The inset displays examples of 4-second snapshots of 
activity in a disconnected network (grey), a random network (black, 
K — 0), and a maximally (though not fully) antisymmetric network 
(orange, k = 1). The spectral abscissa was set to R' = 0.9. Traces 
were obtained from a direct simulation of Eq.[l] and are shown here 
only for two randomly chosen neurons. 



versus inhibitory nature of the presynaptic neuron. Whether 
a connection exists (non-zero entry) is decided as follows. 
Connection Wij with / > J exists with probability p. If / ^ j, 
the reciprocal connection wj; then exists with probability 
p + Cij{l — p) if W/j exists too, or with probability p(l — Cy) if 
it does not. In comparison to the random networks consid- 
ered above (Eq. [3| , this connectivity scheme preserves the 
mean weight w = (wij) — zLpwo/VN as well as the weight 
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variance ((iVy — iv)^) = p(l — p)wq/N while giving full con- 
trol over their normalized covariance c//. Note that c,j can 
assume positive values as high as Cmax = 1, in which case all 
connections are bidirectional. However, c cannot go below 
Cmin — ^P/(l ^ P)i which stems from the sparsity condition 
that imposes a certain degree of symmetry in W: because 
both Wjj and wj; are zero with high probability, they will 
often be null together, meaning that they cannot be fully 
anti-correlated. The limit case c — c,nin corresponds to the 
complete absence of reciprocal connections. Since we aim 
at tilting W towards antisymmetry, we choose Cy — KCmin 
when neurons /' and j are of the same type, and c// = KC,nax 
when the two neurons have different types. Thus < k < 1 
parameterises the degree of antisymmetry in W. As can be 
seen in Fig. [6^, increasing k effectively decreases the spec- 
tral abscissa /?' — max>Re(A), although it is designed not 
to affect the overall connectivity "power" ^ which is the 
relevant quantity for amplification. Thus, for a fixed level of 
dynamical slowing (i.e. fixed /?'), antisymmetric connectiv- 
ity matrices can assume larger weight strengths and thereby 
yield stronger nonnormal amplification than their random 
counterparts, as depicted in Fig. |6}d. Finally, note that a 
matrix with k = 1 is not purely antisymmetric (W^ ^ — W). 
In fact, neural connectivity matrices can never be fully an- 
tisymmetric, because of the constraint that neurons can be 
only excitatory or only inhibitory. This is an advantageous 
restriction here, because a fully antisymmetric matrix - just 
like a fully symmetric one - is in fact a normal operator that 
cannot support transient amplification. 



8 Discussion 

The nonnormal nature of the neuronal connectivity could 
play a major role in the functional dynamics of cortical net- 
works. It can allow fast transients to develop along well- 
defined activity motifs stored in the pattern of synaptic ef- 
ficacies. In networks with locally dense connectivity, the 
balance between excitation and inhibition has been shown 
to generate such type of amplification, accordingly termed 
"balanced amplification" [5]. We have revisited this feature 
in sparse balanced networks in which any two neurons are 
connected randomly with some probability. Random net- 
works had already been studied in terms of their pseudospec- 
trum which only provides bounds on amplification. We 
have chosen a more direct approach and assessed nonnor- 
mality in terms of its functional impact in networks driven 
by stochastic external input. We have explicitly calculated 
the strength of the activity fluctuations that can only be at- 
tributed to the nonnormality of the recurrent connectivity. 
We found nonnormal amplification to be very weak, con- 
cluding that the only way to obtain large amplification in 
random networks is to allow for significant dynamical slow- 
ing. If the dynamics are to be kept fast, then the connectiv- 
ity needs some structuring, so as to allow synaptic weights 
to take up larger values and to discourage the emergence of 
large positive eigenvalues. We have given an example of min- 



imal network structure, namely connection antisymmetry, 
that achieves precisely this. More adaptive ways of shaping 
the connectivity, such as synaptic plasticity, could also be 
considered. In particular, inhibitory synaptic plasticity has 
recently been shown to suppress the attractor dynamics of 
a few activity motifs embedded in a spiking network, while 
still permitting their transient recall |12| . 

Nonnormal amplification could provide a mechanistic ac- 
count for the often reported transient nature of both spon- 
taneous and evoked activity in primary sensory cortices. 
Moreover, from a functional viewpoint, amplification with- 
out slowing could be a highly relevant feature in areas in- 
volved in the processing of fast-changing signals. If transient 
amplification by the synaptic connectivity is meant to allow 
past experience to be reflected in the responses to sensory 
stimuli (see e.g. [14)). then it is quite reassuring that ran- 
dom networks are poor ampliflers, for it implies that nothing 
can be amplified that has not been learned. 

Here we have focused on spontaneous activity, i.e. on the 
fiuctuations elicited by isotropic external noise that is to- 
tally uninformed of the frozen structure of the connectivity 
matrix. The equivalent triangular form of a nonnormal con- 
nectivity matrix suggests that neuronal networks should be 
more sensitive along some input directions than along oth- 
ers, so they could still respond vigorously though transiently 
to some carefully chosen input patterns (evoked activity). 
The first Schur mode, for example, is indeed such a pre- 
ferred pattern [6]. This anisotropy prompts two important 
questions. First, how many different (orthogonal?) direc- 
tions of high sensitivity does a network possess? Similarly, 
in how many distinguishable directions can the network am- 
plify those preferred input signals? These quantities taken 
together could define the "nonnormal information capacity" 
of a network, reminiscent of the concept of memory capacity 
in attractor networks. 

Finally, our analysis has revealed that the nonnormality 
of balanced networks is to a large extent reflected in large 
"DC" fluctuations. This seems to be a general feature of 
networks in which neurons can either be excitatory or in- 
hibitory, but not of a mixed type |15| . It is somewhat dis- 
appointing that however strong activity fluctuations are in 
individual neurons, they always comprise a finite fraction of 
common variability. This is because the variance of the over- 
all population activity is of the same order as the activity 



variance of the individual neurons (Eq. 16 1. Should compu- 
tations exploit the fluctuations along the remaining N — 1 
degrees of freedom of the network, complications in decoding 
the current network state would most certainly arise from 
a single dimension dominating the dynamics. However, we 
wish to point out that these large DC fluctuations are in fact 
a direct consequence of the exact excitation-inhibition bal- 
ance considered here. We show in Appendix[C]that when in- 
hibition dominates over excitation, the variance of the pop- 
ulation activity becomes suddenly inversely proportional to 
the network size. Furthermore, the mean pairwise corre- 
lation coefficient in the network scales similarly, and thus 
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vanishes in large networks unless the E-I balance is exact. 
Note that this phenomenon is not mediated by a destruc- 
tion of the strong feedforward link from the global balance 
disruption d onto the DC mode v, as described at the end of 
section |3] Increasing the overall amount of inhibition does 
preserve this strong link, but cancels its amplifying effect 
by imposing an equally strong negative feedback from the 
DC mode onto itself (see Appendix [C|. This dynamic can- 
cellation of fluctuations and correlations was already shown 
to arise in balanced networks of spiking neurons |16) . Our 
results obtained for linear networks therefore suggest it may 
be a very general feature of inhibition-dominated balanced 
networks, and that fine-tuning the balance until it becomes 
exact [12] may strongly affect the dynamics of the network 
and the resulting correlation structure. 
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A Amplification in random triangular 
networks 

In this appendix we derive an exact expression for ampli- 
fication in random strictly triangular networks with linear 
stochastic dynamics as in Eq.[T] where the non-zero elements 
of the coupling matrix T are drawn from an arbitrary dis- 
tribution with zero mean and variance o? jN where A/ is the 
network size. Though no closed-form solution is known for 
the zero time lag covariance matrix Z, we know from the the- 
ory of multidimensional Ornstein-Uhlenbeck processes that 
it satisfies the so-called Lyapunov equation ITT] 



(T-l)Z + 5: 



(20) 



Equating component (/,_/' < /') on both sides of Eq. 20 yields: 
/-I , j-i 

> (21) 



1 '"^ 1 ' 



k=\ 



k=l 



and equating the diagonal term (/, /') on both sides gives the 
variance of Schur mode /: 



fT// = — + 2^ t;j(Jij 
i=l 



Combining Eqs. [21] and [22] yields 



i-i //-I 



^ + ^ E ^'V ( E ''^"j^ + E 'J^" 



ik 



(22) 



(23) 



in which ajk and aik are to be recursively obtained from Eq. 



21 with proper replacement of indices. We would like to cal- 



culate the expected value over the tij coefficients, i.e. over 
multiple realisations of random matrix T. Explicitly ex- 
panding the sums will reveal cross-terms like (tijtki). Those 
vanish \i i ^ k or j ^ I, because the coupling coefficients are 
taken to be uncorrelated. The only remaining terms will be 
powers of the variance j N. Here we seek a truncation to 
order /N^. Let us calculate: 



/-I ;-i 



^'^^{^^•k^jk) 



(24) 



j=i k=i 

+ 2EE(^f^^'^''<) 

j=l k=l 

Because the network of Schur modes is purely feedforward, 
the cross-covariance ajk for {j, k) < i is independent of the 
coupling coefficients tjj and t/k, thus {tjjtikOjk) — {tij'tik) (o'jk)- 
The only non- vanishing term in the first double-sum is there- 
fore obtained for k — j, giving 



a' 



j=i 



2EE(^^''^j''^*) 

_/=! k=l 



(25) 



Let us expand the expression in the second double-sum using 
Eq.[2l} 

1 '"^ 

( tij tjk c^//c > = 2 51 ( ^'J ^J'' ^' 

k-l 



(26) 



1 

e=i 



{tijtjktkecTh 



As above, the first sum vanishes except for £ — j. Should 
one continue and expand the second sum, one would receive 
terms of order a^/ and more which are discarded here 
(see above). Hence 

^,2 



\tijtjka-ik) 



2N 



\^jk<yjk) 



Using similar arguments, we expand {tjkO'jk] 
and receive: 



a 



'\tjk<yjk) = ^ {(^kk) 



(27) 

to order / N 
(28) 



From Eq. [25] it therefore follows that 



2 '-1 

a \ ^ 



2N ^ 

i=i 

4 /-I J-1 

° EE 

=1 k=\ 



(29) 



8A/2 



'^kk) 



Defining f; — 2 {a;j) /((T|r), we end up with a recursive equa- 
tion for the build-up of relative variance down the feedfor- 
ward network of Schur modes: 



j=i \k=i 



k=l 



a 

2N 



E^- 



a 

SAP 



/-I j-i 

EE^^ 

j=l k=l 



(30) 
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Now we define x — i/N (thus < x < 1) and rewrite Eq. 30 where Q = (2/()!/ [k^ik + 1)!] is the k^^ Catalan number, 
as Assuming g(x) can be written for < x < 1 as a convergent 

power series 



xN 



2 '-1 

a^x \ - 

1 + 



i=i 

ax X - X ^ 

J=l k=l 



(31) 



g:(x) = Jim J2f3,x' 



(37) 



/(=0 



and equating '{0) in both Eqs.|36|and 37 yields the results 



of Eqs. 13 -p 



In the limit A/ — oo with constant x — i / N ratio, the sums 
on the r.h.s. converge to their corresponding Riemann inte- 
grals, endowing ^a/ with a proper limit g{x): 



OL^X^ 



[ ds f ds'Q{s- s')g{xs') 
Jo Jo 



(32) 



where is the Heaviside function. This convergence stems 
from the 1/N scaling of the variance a^/N. Using straight- 
forward changes of variables (s i— > s/x), we end up with an 
integral equation for g, the continuous variance profile along 
the (now infinitely large) network of Schur patterns: 

g (x) = 1 + - g (s) ds + - ds ds'g (s') (33) 

Differentiating Eq. [33] twice with respect to x yields a 
second-order differential equation for g 



B Variance of the DC component 

The last Schur mode is fed by the activities of all previous 
Schur vectors, weighted by couplings with variance Qq/N. 
The same calculation that led to Eq. [HOlin this case leads to 



fN = 1 



A/-1 

i=l 



.2p2 N-lj-l 



8A/ 



(38) 



_/■=! k=l 



which can be rewritten as 
N 




(39) 



where the sums were previously calculated in the limit N 



oo (Eqs. 30 - 37). We thus recover 



(40) 



With given by Eq. [sjwe arrive at Eq. 16 of the main text 



fix) 



-g' {x) + ^g{x) 



(34) C Exactly balanced vs. inhibition-dominated 

networks 



with initial conditions g(0) — 1, g'(0) — and g"(0) 
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of 



3a^/8. The solution is precisely g^^{x) given in Eq, 
the main text. It is only a lower-bound on the true variance 
profile g(x) since all the higher-order terms in that we 
have neglected are positive. This approximation proves rea- 
sonable for < 3 as shown in Fig. [3^ (dashed blue lines). 
Further integrating over x yields a lower-bound on nonnor- 
mal amplification /4o(a^) = g (x) dx — 1 (Fig. [ijb, dashed 
blue line): 




(35) 



Instead of truncating {an) to order a^, one can also decide 



to start again from Eq. 24 and keep all terms up to order n. 
This requires careful counting, and results in a differential 
equation of order n, reading 



k=0 



a 



Jn-k-l) 



{x) 



(36) 



In this paper, we have considered connectivities in which 
weights were either zero or ±wq/\/N, the ± sign depending 
on the excitatory vs. inhibitory nature of the presynaptic 
neuron (Eq. [S]) . Furthermore, the number of cells of both 
types was identical. The total inhibitory synaptic strength 
thus exactly matched its excitatory counterpart. In this 
appendix, we wish to show that if the non-zero inhibitory 
weights are stronger, i.e. —jwo/VN with 7 > 1, the dynam- 
ics of the overall population activity is strongly affected. 

We have seen that the "DC" mode v = (1 1)/^/N is an 

eigenvector of W. Let Ai, denote the associated eigenvalue, 
which quantifies the effective decay rate of the DC compo- 
nent in the network of neurons. If the E-I balance is exact 
(7 = 1) as assumed throughout the paper, then A^, = 0. 
More generally, however, one can calculate 

A. = _ P-o(7-l) .V,v (41) 

We see there is an unexpected scaling that the exact balance 
was hiding : — A^ ^ ©(-vZ/V). Note that all other eigenvalues 
are now scattered inside the disk of radius 



R 



Wo 



(l+72)p(l-p) 



(42) 
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though no longer uniformly so since the variance of the in- 
hibitory and excitatory weights now differ by a factor of 7^ 
|10j . Having kept the focus of this paper on nonnormal ef- 
fects, we have intentionally set aside the contributions of 
the eigenvalues to the overall amplification in the network. 
When Ai, = (perfect balance), our prediction that the av- 
erage population activity = J2^ii^)/^ should have a 
variance of order was justified : the last Schur unit 

corresponding to this DC indeed receives N — 1 contribu- 
tions of order 0(1), and its decay time constant is simply 
T ~ 0(1), yielding var[/i(t)] ^ 0{1). When inhibition dom- 
inates (7 > 1), the DC component suppresses itself via a 
negative feedback that scales with a/A/, yielding a very short 
decay time constant r/(l — X^,) ^ 0{1/VN) whose devia- 
tion from r can no longer be neglected. To see what the 
implications of this scaling are for the variance of /i(t), let 
us reduce the dynamics of the DC to the following set of N 
stochastic differential equations: 



oy; = Vi 

T 



d^i for 1 < / < A/ 



dy^ 



(43) 



Here yi, ... .yw-i model the first N — 1 Schur units inde- 
pendently, with the appropriate noise terms such that they 
achieve a variance of one (corresponding to the limit of small 
amplification) . They feed y^ - which models the activity of 
the last Schur unit, i.e. the DC component ii{t)^fN - with 
couplings Ei such that 'Y^sj/N — Cq. We calculate the cou- 
pling variance the same way we did in section [3] 



p'^o'(l + 7') 



(44) 



The variance var[/i(t)] of the overall neuronal population 
activity, here modeled by /i(t) w yA/(t)/\/A/, is given by 
standard Ornstein-Uhlenbeck theory: 
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Figure 7: Temporal fluctuations of the overall population firing rate 
in a balanced neuronal network. The variance of the average pop- 
ulation activity /i(t) = X]-^i(f)/'^ reported as a function of the 
network size A/, in logarithmic scale. When inhibition perfectly bal- 
ances excitation (7 = 1), the variance is asymptotically independent 
of the network size (gray). When inhibition dominates (7 > 1), it 
scales with 1/A/ (black). The solid lines denote the approximation in 



45 



The dashed lines indicate the asymptotics (Eq. 46 1. Points 
denote the empirical variance obtained by simulating Eq. [Ij for 100 
seconds, for a neuronal network constructed as specified in section 
[Sjwith connectivity density p = 0.1. The spectral radius was set to 
R = 0.1 (top plot) and R = 0.5 (bottom plot). 



var(^(t)) 



1 



(45) 



Although we have neglected amplification and correlations 
in the first N ~ 1 Schur units, Eq. 45 does provides a 
good intuition for how the mean population activity fi{t) — 
yi\i{t)/y/N scales with the network size N, and provides a 
good qualitative match to numerical results even for a non- 
negligible spectral radius /? = 0.5 (Fig. [7|. 
The asymptotics of var[/.t(t)] are given by 



2 2 



2(1 + 7^) 
A/(7 - 1)2 



if 
if 



7=1 
7 > 1 



(46) 



Thus, when inhibition dominates over excitation (7 > 1), 
the fluctuations of the overall population activity vanish 
for large networks, which was already shown in [16] for 



inhibition-dominated networks of spiking neurons. In con- 
trast, fine tuning the connectivity such that the balance be- 
comes exact (7 = 1) opens the possibility for these fluc- 
tuations to subsist in arbitrarily large networks. This has 
profound consequences for the mean pairwise correlation 
7 = X^/^j cov[x;(f-), Xj(t)]/A/2, as seen from the following 
identity 



r = var[^(t)] - -p^var[x,(t)] 



(47) 



We have seen that the average variance var[x/(t)] in the 
individual neurons (i.e. amplification as we define it) is 



0(1). Thus, Eq. 47 implies that r scales with N in the same 



way var[/i(t)] does: either 0(1) if the balance is perfect, or 
0(1/A/) if inhibition dominates. 
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